Scalable network monitoring system

ABSTRACT

A scaleable network monitoring system is discussed. The network monitoring system identifies network monitoring information for the network elements being monitored. By storing only a non-redundant subset of the identified network information in memory, the network monitoring system is able to monitor a much larger group of network elements than is possible with conventional monitoring systems which are burdened by memory constraints. The scaleable network monitoring system also employs a multi-threaded architecture that dynamically spawns an array of multi-technology monitoring sub-systems.

RELATED APPLICATION

This application is a continuation application claiming priority to U.S.patent application Ser. No. 12/129,914 entitled “Scalable NetworkMonitoring System” filed on May 30, 2008, now U.S. Pat. No. 9,003,010,which claimed the benefit of U.S. Provisional Patent Applicationentitled “Scalable Network Monitoring System” filed on May 30, 2007,Application No. 60/940,838, the contents of both applications beinghereby incorporated by reference herein in their entirety.

FIELD OF THE INVENTION

The embodiments of the present invention relate generally to themonitoring of physical and logical elements of a computer network, andmore specifically to the composition and use of a scalable monitoringsystem used to monitor network elements.

BACKGROUND

In order to monitor a physical or logical element on a network, acertain amount of information must be known about the element, includingdetails about the location of the agent or device upon which the networkelement is located and information on how to access the agent or device.In addition, further information may be needed as to the specific groupsof information to request and which instances of those groups torequest. The groups of information may be separated into threecategories: agents, access methods and type of device being monitored.In conventional monitoring systems, the three categories of informationare included with a listing of the item being monitored by the networkmonitoring system.

BRIEF SUMMARY

Embodiments of the present invention provide a scaleable networkmonitoring system. The network monitoring system identifies networkmonitoring information for the network elements being monitored. Bystoring only a non-redundant subset of the identified networkinformation in memory, the network monitoring system is able to monitora much larger group of network elements than is possible withconventional monitoring systems which are burdened by memoryconstraints. The scaleable network monitoring system also employs amulti-threaded architecture that may dynamically spawn an array ofmulti-technology monitoring sub-systems. The subsystems may bedynamically spawned in the network monitoring system's host device(s)and the devices being monitored based on the configuration of thenetwork monitoring system and the maximum possible load that can beplaced on the available Network Interface Cards (NICs).

In one embodiment, a method for scalable network monitoring includes thestep of providing a network monitoring facility. The network monitoringfacility monitors multiple network elements that are communicating overa network and identifies a collection of network monitoring informationfor each of the network elements. The method also selectively stores inmemory a non-redundant subset of the network monitoring information forthe network elements.

In another embodiment, a system for scalable network monitoring includesa network monitoring facility that monitors a plurality of networkelements that communicate over a network. The system also includes acollection of network monitoring information for each of the networkelements being monitored that is identified by the network monitoringfacility. The system additionally includes a non-redundant subset of thenetwork monitoring information for the network elements that has beenselectively stored in memory.

BRIEF DESCRIPTION OF THE DRAWINGS

The invention is pointed out with particularity in the appended claims.The embodiments of the present invention may be better understood byreference to the following description taken in conjunction with theaccompanying drawings, in which:

FIG. 1 depicts an environment suitable for practicing an embodiment ofthe present invention;

FIG. 2 depicts a block diagram of an exemplary network monitoringfacility employed by an embodiment of the present invention;

FIG. 3 depicts a block diagram of an exemplary referencing systememployed by an embodiment of the present invention to store only aselective subset of the identified network monitoring information;

FIG. 4 depicts a block diagram showing the use of managers and resourcepools by an embodiment of the present invention;

FIG. 5 depicts a block diagram showing the ability of an embodiment ofthe present invention to scale across multiple devices; and

FIG. 6 is a flowchart of an exemplary sequence of steps that may befollowed by an embodiment of the present invention to provide ascaleable network monitoring system.

DETAILED DESCRIPTION

Conventional network monitoring systems include information related toeach item being monitored with a listing of the items being monitored.Unfortunately, this inclusion of the related information with theidentified network element leads to a significant amount of unnecessaryduplication of information. This duplication of information in thememory of the device hosting the conventional network monitoring systemsleads to a practical constraint on the amount of network elements thatcan be monitored by a network monitoring system and prevents networkmonitoring and monitoring stations from scaling to monitor massiveamounts of network elements.

The embodiments of the present invention provide a mechanism addressingthis drawback of unnecessarily duplicated information that is performedby conventional monitoring systems. In contrast to conventionalmonitoring systems, embodiments of the present invention avoid theunnecessary duplicate recording of network monitoring information andalso dynamically spawn monitoring tasks appropriate for the number ofelements being monitored. As a result, the embodiments of the presentinvention are able to efficiently scale to monitor networks with anumber of elements that are an order of magnitude larger than thosepreviously able to be monitored by conventional network monitoringsystems.

FIG. 1 depicts an environment suitable for practicing an embodiment ofthe present invention. A computing device 10 hosts a network monitoringfacility 12. The network monitoring facility 12 is an executablesoftware process or processes capable of monitoring large numbers ofphysical and/or logical network elements. The computing device 10 may bea PC, workstation, server, laptop, mainframe, PDA or other computingdevice equipped with a processor and capable of hosting the networkmonitoring facility 12 described herein. The computing device 10 is incommunication over a network 20 with network elements 30, 50 and 70. Itshould be appreciated that the display of only three network elements ismade for reasons of ease of illustration and that a deployed monitoringfacility 12 of the present invention may monitor up to millions ofnetwork elements.

The network 20 may be an internet, the Internet, a Local Area Network(LAN), a Wide Area Network (WAN), a wireless network, an intranet, anextranet or some other type of network. Each network element 30, 50 or70 includes or references network monitoring information 40, 60 or 80which may be used by the network monitoring facility 12 to monitor thecondition and status of the network elements. The network monitoringinformation may include type information 42 about the type of networkelement, agent information 44 about an agent or agents associated withand used to communicate with the network element, and agent accessinformation 46 about the methods used to access the agent. It will beappreciated that additional types of information associated withmonitoring the network element may also be provided. It should also berecognized that the arrangement of system components depicted in FIG. 1is meant as an illustrative example of one of many possible environmentssuitable for practicing the embodiments of the present invention.

FIG. 2 depicts the exemplary network monitoring facility 12 in greaterdetail. The exemplary network monitoring facility 12 includes an agentconfiguration module 201, an agent access configuration module 202 andan element configuration module 203 respectively holding agentinformation, agent access methods and element information for networkelements being monitored. The agent configuration module 201 may holdbasic information relating to the agents and/or devices that are beingmonitored (e.g. IP Address). The agent access configuration module 202may hold more specific information about how that agent is accessed(e.g. instructions to use the SNMP protocol at UDP port 161 withcommunity string “public”, or connect to TCP port 1152 with username“mysql” and password “mysql”). The element configuration module 203 mayhold unique information that pertains to each of the elements beingmonitored and that is not described in either the agent configurationmodule 201 or the agent access configuration module 202.

The network monitoring information that is stored in the agentconfiguration module 201, agent access configuration module 202 andelement configuration module 203 may be received by the networkmonitoring facility 12 in multiple ways. For example, the networkmonitoring facility 12 may read a file, listen for updates on a TCPport, or connect to a database to identify network monitoringinformation necessary to monitor a network element. As the networkmonitoring facility 12 identifies and receives network monitoringinformation it may update the agent configuration module 201 by adding,removing, or updating an agent item object 205 in the agentconfiguration module. An agent item object 205 is an object thatcontains basic information about the agent such as addressinginformation (IP address, hostname, location, etc), statistics onavailability and reachability, protocol based timeouts, and may containa set of references to agent access item objects 204.

The network monitoring facility 12 may also update the agent accessconfiguration module 202 by adding, removing, or updating an agentaccess item object 204. An agent access item object may include a set ofinformation describing how to access the agent including but not limitedto the network protocol and/or ports to use, and any specific accessinformation like community strings, usernames, passwords, etc.References to agent item objects 205 may be stored in agentconfiguration module 201 in a list or other data structure. The agentaccess configuration module 202 may also contain a list or other datastructure referencing agent access item objects 204. Each agent itemobject 205 can optionally include a list 206 of agent access types thatare references to agent access item objects 204. Multiple agent itemobjects 205 may include references to the same Agent Access Item object204 if the access methods are the same.

The element configuration module 203 may include a monitored elementslist 207. The network monitoring facility 12 may add, remove, or updatethe monitored elements list 207 in the element configuration module 203.When the network monitoring facility 12 adds, removes or updates themonitored elements list 207, each element being monitored is assigned toan agent item object 205 and an Agent Access Item object 204. If amonitored element references an Agent that has not been defined in theagent configuration module 201 or an Access method that has not beendefined in the agent access configuration module 202, a new agent itemObject 205 or Agent Access Item 204 will be created and added to theirrespective configuration modules.

The interrelationship between the monitored elements 207, agent itemobjects 205 and agent access item objects 206 is depicted in FIG. 3. Ascan be seen in FIG. 3, multiple Monitored Element Items 301, 302 and 303can point to the same agent item object 304 and Agent Access Item object306. Similarly, an agent item object 304 can point to multiple AgentAccess Item objects 305 and 306. As an example, Monitored Element Item 1(301) points at the agent item object 304 and the Agent Access Itemobject 1 (305). Monitored Element Item 2 (302), points at the agent itemobject 304 and the Agent Access Item object 2 (306). Monitored ElementItem 3 (303) points at the agent item object 304 and the Agent AccessItem object 2 (306). Since all monitored element Items 301, 302 and 303point at agent item object 304, it can be inferred, by the agentconfiguration module 201 or other process in the network monitoringfacility 12, that agent item object 304 could point to both Agent AccessItem object 1 (305), and Agent Access Item object 2 (306). As a result,the network monitoring facility 12 only needs store one copy of thisobject in the agent configuration module 201 and each Monitored ElementItem only needs one reference to the same object.

The non-duplicative storage of monitoring information provides asignificant memory savings. The storage of a single object may take up aconsiderable amount of memory, but a reference to the object takes uponly the memory needed to store the reference (e.g. 4 bytes). It is thissharing of information through references to similar objects that allowsthe network monitoring facility 12 to save enormous amounts of space inthe random access memory (RAM) on the computing device 10 that hosts thenetwork monitoring facility. The saving of space in the RAM allows thestorage of configuration information for literally millions of monitoredelements. The network monitoring facility 12 is not forced to read allthe available monitoring information as it is needed from a databasebecause of the efficient storage of configuration and access informationused in RAM. Likewise, if database storage is needed, the networkmonitoring facility 12 can also store this information in the database.For example, in one embodiment, a table for agents, a table for uniqueaccess methods, and another table for monitored element items may beused. Each row in the monitored element item table may have at least onecolumn that refers to a row in the agents table, and one column thatwould refer to a row in the agent access methods table. A networkmonitoring facility 12 can also store this information on disk in afile, using a section to define agents, a section to define accessmethods, and another section to define monitored elements, each of whichhave a reference to an agent and an access method.

The network monitoring facility 12 is also able to prevent contentionfor the same hardware or software resources. The contention is dealtwith on three levels—resources within the software application, theresources of the agents being monitored, and the resources of themachine on which the application is running. To remove contention forresources within the software, the network monitoring facility 12 uses aset of managers and resource pools. These “managers” can check for anidle resource, and, if one is available, provision the idle resource. Anidle resource may be a polling engine, results collector, output engine,network interface card, etc. Each resource has a concept of “state”. Anidle resource is provisioned by changing the state of the resource to astate other than “idle” (i.e. initializing, ready, or busy), and passinga reference to the resource back to the requestor. In the event thereare no idle resources, a new resource is added to a resource pool and atask is assigned to the new resource.

The use of the resource pool in dynamically allocating resources toperform network monitoring is depicted in FIG. 4. In one embodimentdepicted in FIG. 4, the network monitoring facility 400 launches ascheduled task, referred to herein for illustrative purposes as aMonitorDispatcher task 401. The MonitorDispatcher task 401 determineswhich Monitored Elements 207 should be monitored based on the currentstate of the element and the last poll time, and then makes one or morerequests to a Poller Engine Pool 402 for one or more Poller resources403. A poller resource 403 is a process that iterates through themonitored elements and, using technology-specific poller blades 404,makes the queries over the network. The poller blades 404 are objectresources that are created and used by the poller object. Each pollerblade uses a specific technology for data acquisition. One can be SNMP,another XML/SOAP, another FTP, another may use TCP connects, etc. Thepoller blades 404 are used by the poller 403 when there are monitorsthat need data collected using that specific technology. Otherwise, theyare idle and kept in a resource pool. In one embodiment of the presentinvention, the PollerBlades 404 may perform the actual access andcollection of data while the Poller 403 is actually just another“manager”. This represents a departure from conventional systems wherethe poller 403 would do the actual collection of data, and would not becapable of provisioning other pollers or querying agents in differentways (i.e. using different protocols and access methods).

The amount of poller resources spawned to perform network monitoring isdetermined based on a number of factors including the number of elementsto be monitored, tuning information that may have been specified in theoriginal startup configuration of the network monitoring facility 12,and the response times from the agents being polled. If there is staticinformation in the startup configuration concerning the default numberof blades per poller, the default number of pollers per pool, and/orwhich network interface cards can be used for monitoring, thatinformation may be used as a base configuration for tuning pollerresources. In addition, if the system so determines that the network isnot responding quickly enough to monitor the number of elements in theconfiguration, and there is a large backlog of requests, the number ofpollers may be dynamically increased and the requests more evenly spreadout among those pollers and any available network cards in order toincrease the overall bandwidth of the monitoring process. The system mayalso automatically determine if any one network card is overloaded withtraffic, and if so, may round-robin requests with other network cardsthat are not as busy. Additionally, the system may also manage resourcesfor slow responders by interleaving those requests with fasterresponders, thereby eliminating the possibility for any one particulardevice to cause a bottleneck in the polling process.

The poller resource 403 may be assigned a specified time window to queryall the elements and send the results to the Monitor Results Manager405. The MonitorDispatcher task 401 may then dispatch some or all ofits' scheduled monitored elements 207 to the provisioned Pollers 403.Once provisioned, a Poller 403 may inform a Monitor Results Manager 405that it needs a Results Collector object 406. The Monitor ResultsManager 405 may create a new Results Collector object 406 and assign itto the Poller 403. The Poller resource 403 may then determines what typeof Poller Blade 404 should be provisioned according to the number ofMonitored Elements 207 and access the types of Agent Access Item objects204 to which the Monitored Elements refer.

Once results are collected by the Results Collector object 406, theMonitor Results Manager 405 communicates with a Results Analysis EnginePool 407 and retrieves an idle or new Results Analysis Engine object408. The Results Analysis Engine object 408 processes the collectedresults and then creates a new Output Engine object 409, which outputsthe results in a desired format. The output may be directed to a disk,to a database, to a screen, etc. Once each “manager” finishes its'portion of the task, it is returned to the pool from whence it came. TheResults Analysis Engine object 408 is returned to the Results AnalysisEngine Pool 407, and the Poller 403, is returned to the Poller EnginePool 402.

The network elements can be polled by several different types oftechnologies and protocols (e.g. using SNMP, XML, FTP, ICMP, POP3, TCPconnect, etc). An example of polling with SNMP would be querying anagent for the objects sysUpTime, ifOperStatus, ifInOctets, IfSpeed.Using the results from these objects one can determine the amount oftime the network element has been available, its' current status, andthe bandwidth utilization of the interface. The object ifInOctets isrepresented by the Object Identifier (OID) 1.3.6.1.2.1.2.2.1.10. To pollthis object for interface 1 a poller using SNMP may send a SNMPGET querywith the OID 1.3.6.1.2.1.2.2.1.10.1. The returned result is a large32-bit integer. The value returned may be stored by the networkmonitoring facility 12 and the difference between the values on thecurrent poll and the last poll may be used for analysis.

If it is determined that a monitored agent is not responding in anadequate time period, the network monitoring facility may slow down therequests to that particular agent and interleave requests to the sloweragent with those to faster responding agents. In this way the overallmonitoring process is not slowed down and a large number of networkelements can still be monitored in the allotted time frame.Additionally, when the configuration on monitored elements continues togrow, the network monitoring facility can continue to dynamically spawnadditional poller resources 403, each of which itself can then expandthe number of thread resources being used. The network monitoringfacility can continue to spawn additional pollers, based on the numberof total available threads the computing device supporting themonitoring facility can handle. The dynamic decision as to the number ofthreads which the network monitoring facility can utilize in spawningadditional polling resources is based on comparing the number of threadssupportable by the number of CPUs (and cores in those CPUs) for thecomputing device compared with the number of threads being currentlyutilized.

When a network monitoring facility needs to scale to monitor largernumbers of network elements, it may start to contend for resources onthe computing device upon which it is running. To address this issue,the network monitoring facility is capable of scaling across multiplemachines. For example, while the network monitoring facility mayfunction as a stand-alone process it is also capable of communicatingwith other systems being provisioned by a central network monitoringfacility such as the central network monitoring facility depicted inFIG. 5. A first monitoring facility 501 can be told by a central networkmonitoring facility 502, that it needs to monitor a certainconfiguration of network elements 503. Another network monitoringfacility 504 can be told by the same central network monitoring facility502 that it needs to monitor a certain configuration of network elements505. Each network monitoring facility is assigned an identificationnumber at startup, and polls network elements from the centralconfiguration based on this unique identification number. Once thenetwork monitoring facilities 501 and 504 are finished with their pollcycles, they can output the results to some type of storage like adatabase 506 or files on disk 507. Alternatively, the results may bestreamed directly back to the central network monitoring facility 502.The embodiments of the present invention thus allow a number of networkmonitoring facilities to be running locally on a network or in adistributed fashion over a geographic and/or logical area, allmonitoring their own subset of the entire network monitoringconfiguration.

FIG. 6 is a flowchart of an exemplary sequence of steps that may befollowed by an embodiment of the present invention to provide ascaleable network monitoring system. The sequence begins with theproviding of a network monitoring facility (step 600). The networkmonitoring facility identifies network information for network elementsthat are to be monitored (step 602). A selected subset of the availablenetwork monitoring information is retrieved (step 604) and stored inmemory (step 606). The ability of the network monitoring facility tofunction and only store a subset of the available network monitoringinformation in memory enables the scaling of the network monitoringfacility to track extremely large numbers of network elements. Once thenetwork monitoring information has been stored, the network monitoringfacility monitors the associated network elements (step 608).

The present invention may be provided as one or more computer-readableprograms embodied on or in one or more mediums. The mediums may be afloppy disk, a hard disk, a compact disc, a digital versatile disc, aflash memory card, a PROM, a RAM, a ROM, or a magnetic tape. In general,the computer-readable programs may be implemented in any programminglanguage. Some examples of languages that can be used include FORTRAN,C, C++, C#, or JAVA. The software programs may be stored on or in one ormore mediums as object code. The code may run in a virtualizedenvironment such as in a virtual machine. Multiple virtual machinesrunning the code may be resident on a single processor.

Since certain changes may be made without departing from the scope ofthe present invention, it is intended that all matter contained in theabove description or shown in the accompanying drawings be interpretedas illustrative and not in a literal sense. For example, throughout thedescription herein, reference has been made to named types of objects.It should be appreciated that the naming of the types of objects is donefor convenience in explaining the present invention to the reader andthat objects with similar functionality but different names are also tobe considered within the scope of the present invention. Similarly,practitioners of the art will realize that the sequence of steps andarchitectures depicted in the figures may be altered without departingfrom the scope of the present invention and that the illustrationscontained herein are singular examples of a multitude of possibledepictions of the present invention.

I claim:
 1. A non-transitory medium holding computer-executableinstructions for performing scalable network monitoring that whenexecuted by a computing device: provide a network monitoring facilitythat includes a plurality of monitoring resources, the networkmonitoring facility monitoring a plurality of network elementscommunicating over a network with the plurality of monitoring resources;dynamically spawn at least one additional monitoring resource based on aresponse time of at least one of the plurality of network elements inresponding to a request initiated by the network monitoring facility,the at least one additional monitoring resource provisioning and usingan idle technology-specific object resource from a pool of idle objectresources to perform monitoring of at least one network element; anddynamically adjust a pattern of monitoring requests based on a responsetime of at least one of the plurality of network elements in respondingto a request initiated by the network monitoring facility, the adjustinginterleaving a request to a slower responding network element withrequests to faster responding network elements.
 2. The medium of claim 1wherein the medium further holds instructions that when executed by acomputing device: identify a collection of network monitoringinformation for each of the plurality of network elements with thenetwork monitoring facility; and selectively store in memory anon-redundant subset of the collection of network monitoring informationfor the plurality of network elements.
 3. The medium of claim 1 whereinan amount of resources spawned is based on a number of Network InterfaceCards (NICs) available for use in performing the monitoring of theplurality of network elements.
 4. The medium of claim 1 wherein anamount of monitoring resources spawned is based on a number of threadsavailable on a computing device supporting the monitoring facility. 5.The medium of claim 1 wherein an amount of monitoring resources spawnedis based on the response time of the at least one network element inresponding to the request initiated by the network monitoring facility.6. A computer-implemented method for performing scalable networkmonitoring, comprising: providing a network monitoring facility thatincludes a plurality of monitoring resources, the network monitoringfacility monitoring a plurality of network elements communicating over anetwork with the plurality of monitoring resources; spawning dynamicallyat least one additional monitoring resource based on a response time ofat least one of the plurality of network elements in responding to arequest initiated by the network monitoring facility, the at least oneadditional monitoring resource provisioning and using an idletechnology-specific object resource from a pool of idle object resourcesto perform monitoring of at least one network element; and adjustingdynamically a pattern of monitoring request based on a response time ofat least one of the plurality of network elements in responding to arequest initiated by the network monitoring facility, the adjustinginterleaving a request to a slower responding network element withrequests to faster responding network elements.
 7. The method of claim 6further comprising: identifying a collection of network monitoringinformation for each of the plurality of network elements with thenetwork monitoring facility; and selectively storing in memory anon-redundant subset of the collection of network monitoring informationfor the plurality of network elements.
 8. The method of claim 6 whereinan amount of resources spawned is based on a number of Network InterfaceCards (NICs) available for use in performing the monitoring of theplurality of network elements.
 9. The method of claim 6 wherein anamount of monitoring resources spawned is based on a number of threadsavailable on a computing device supporting the monitoring facility. 10.The method of claim 6 wherein an amount of monitoring resources spawnedis based on the response time of the at least one network element inresponding to the request initiated by the network monitoring facility.11. A computing device for performing scalable network monitoring,comprising: a network interface, and a processor configured to executeinstructions that: provide a network monitoring facility that includes aplurality of monitoring resources, the network monitoring facilitymonitoring a plurality of network elements communicating over a networkwith the plurality of monitoring resources; dynamically spawn at leastone additional monitoring resource based on a response time of at leastone of the plurality of network elements in responding to a requestinitiated by the network monitoring facility, the at least oneadditional monitoring resource provisioning and using an idletechnology-specific object resource from a pool of idle object resourcesto perform monitoring of at least one network element; and dynamicallyadjust a pattern of monitoring request based on a response time of atleast one of the plurality of network elements in responding to arequest initiated by the network monitoring facility, the adjustinginterleaving a request to a slower responding network element withrequests to faster responding network elements.
 12. The computing deviceof claim 11 wherein the processor is also configured to executeinstructions that: identify a collection of network monitoringinformation for each of the plurality of network elements with thenetwork monitoring facility; and selectively store in memory anon-redundant subset of the collection of network monitoring informationfor the plurality of network elements.
 13. The computing device of claim11 wherein an amount of resources spawned is based on a number ofNetwork Interface Cards (NICs) available for use in performing themonitoring of the plurality of network elements.
 14. The computingdevice of claim 11 wherein an amount of monitoring resources spawned isbased on a number of threads available on a computing device supportingthe monitoring facility.
 15. The computing device of claim 11 wherein anamount of monitoring resources spawned is based on the response time ofthe at least one network element in responding to the request initiatedby the network monitoring facility.